Systems and methods for unifying two-dimensional and three-dimensional interfaces

ABSTRACT

This disclosure relates to systems, devices, and methods for unifying a two-dimensional (2D) interface and a three-dimensional (3D) interface of a computer device, with each interface having unique inputs and displays specific to each dimensionality. The present disclosure provides for identifying a display context value that indicates whether a user is viewing a two dimensional (2D) display of a 2D space or a three dimensional (3D) display of a 3D space of the computer device, as well as identifying an input context value that indicates whether a user input received at an input device is directed toward the 2D space or the 3D space of the computer device, and selecting one of a plurality of operating states of the computer device based on the display context value and the input context value.

BACKGROUND

A computer graphics system with a two-dimensional (2D) interface and athree-dimensional (3D) interface will take input and have displaysspecific to each dimensionality. For example, a traditional 2D systemwill have a mouse, a keyboard, and a monitor, and this 2D system can beextended by adding a 3D system that includes a Head Mounted Display(HMD) and a six-degree-of-freedom (6DOF) controller. These systems canoperate in parallel on the same set of applications and data, whichcreates challenges in determining where input should go and what shouldbe shown on which display. Current Virtual Reality (VR) softwaresolutions use a modal model, where only one form of display and inputcan be used at a time. That is, a user may either interact with the 2Dinterface or the 3D interface, such that each interface is mutuallyexclusive.

Thus, there exists a need for improvement in the interoperabilitybetween 2D computer graphics systems and 3D computer graphics systems.

SUMMARY

The following presents a simplified summary of one or moreimplementations of the present disclosure in order to provide a basicunderstanding of such implementations. This summary is not an extensiveoverview of all contemplated implementations, and is intended to neitheridentify key or critical elements of all implementations nor delineatethe scope of any or all implementations. Its sole purpose is to presentsome concepts of one or more implementations of the present disclosurein a simplified form as a prelude to the more detailed description thatis presented later.

One example implementation relates to a method of operating a computerdevice. The method may include identifying a display context value thatindicates whether a user is viewing a two dimensional (2D) display of a2D space or a three dimensional (3D) display of a 3D space of thecomputer device. The method may also include identifying an inputcontext value that indicates whether a user input received at an inputdevice is directed toward the 2D space or the 3D space of the computerdevice. The method may further include selecting one of a plurality ofoperating states of the computer device based on the display contextvalue and the input context value, the plurality of operating statesincluding a 2D input/3D display operating state and a 3D input/2Ddisplay operating state. The method may also include operating thecomputer device according to the selected operating state.

In another example, a computer device for selecting an operating stateis disclosed. The computer device may include a memory to store data andinstructions and a processor in communication with the memory. Theprocessor may be configured to execute instructions to identify adisplay context value that indicates whether a user is viewing a twodimensional (2D) display of a 2D space or a three dimensional (3D)display of a 3D space of the computer device. The processor may beconfigured to execute instructions to identify an input context valuethat indicates whether a user input received at an input device isdirected toward the 2D space or the 3D space of the computer device. Theprocessor may be configured to execute instructions to select one of aplurality of operating states of the computer device based on thedisplay context value and the input context value, the plurality ofoperating states including a 2D input/3D display operating state and a3D input/2D display operating state. The processor may be configured toexecute instructions to operate the computer device according to theselected operating state.

Another example implementation relates to computer-readable mediumstoring instructions executable by a computer device. The instructionsmay include code for identifying a display context value that indicateswhether a user is viewing a two dimensional (2D) display of a 2D spaceor a three dimensional (3D) display of a 3D space of the computerdevice. The instructions may include code for identifying an inputcontext value that indicates whether a user input received at an inputdevice is directed toward the 2D space or the 3D space of the computerdevice. The instructions may include code for selecting one of aplurality of operating states of the computer device based on thedisplay context value and the input context value, the plurality ofoperating states including a 2D input/3D display operating state and a3D input/2D display operating state. The instructions may include codefor operating the computer device according to the selected operatingstate.

Additional advantages and novel features relating to implementations ofthe present disclosure will be set forth in part in the description thatfollows, and in part will become more apparent to those skilled in theart upon examination of the following or upon learning by practicethereof.

DESCRIPTION OF THE FIGURES

In the drawings:

FIG. 1 a schematic diagram of an example computer device having anoperating state controller configured to enable a 2D/3D interoperabilitystate in accordance with an implementation of the present disclosure;

FIG. 2 is a schematic diagram of an example head-mounted display (HMD)in accordance with an implementation of the present disclosure;

FIG. 3 is a flowchart of an example method for operating a computerdevice having interoperable 2D and 3D interfaces in accordance with animplementation of the present disclosure;

FIG. 4 is a schematic block diagram of an example computer device inaccordance with an implementation of the present disclosure; and

FIG. 5 is an example chart illustrating transitions between operatingstates in response to receiving a toggle input in accordance with animplementation of the present disclosure.

DETAILED DESCRIPTION

This disclosure relates to systems, devices, and methods for unifying atwo-dimensional (2D) interface and a three-dimensional (3D) interface ina computer device, with each interface having unique inputs and displaysspecific to each dimensionality. The present disclosure describes acomputer device having an operating state controller that enables one ormore 2D/3D interoperability operating states, or solely a 2D or a 3Doperating state, based on certain display context values and an inputcontext values. In general, the display context value indicates whethera 2D or a 3D display is being viewed by the user, and the input contextvalue indicates to which space a user input is being directed. As such,the operating state controller may determine an operating state to beone of a 2D input/2D display state, a 2D input/3D display state, a 3Dinput/2D display state, or a 3D input/3D display state. As such, thepresent disclosure enables the use of 2D inputs with a 3D display in the2D input/3D display state, and the use of a 3D input with a 2D displayin the 3D input/2D display state, and enables efficient transitioningbetween different operating states based on monitoring display contextand input context values.

In an implementation, for example, the display context value may trackusage of an HMD. For instance, while the HMD is actively worn by theuser, the system may assign a value of a 3D display to the displaycontext value, which may indicate that any new application views shouldbe placed into a 3D space presented by the HMD, rather than being placedin a 2D space (e.g., on a 2D display associated with the computingdevice). In contrast, when the HMD is not worn by the user or is notpresent and/or plugged in, the display context value may indicate thatany new application views should be displayed in the 2D space.

In some implementations, however, the display context value may beoverridden by, for example, a user entering one or more keys. Thisallows the user to manually drive a change of operating states, forinstance, when the user wants to direct a specific input to a specificone of the 2D space or the 3D space.

In further implementations, each window in the system may be assigned adisplay context value indicating where that window is rendered orpresented to the user. For example, if a window has a 2D display contextvalue, then the window may be shown in the 2D space. In contrast, if thewindow has a 3D display context value, then the window may be shown tothe user in the HMD.

In still further implementations, an input context value may trackwhether the user's input focus, e.g., an input into a mouse and/or akeyboard, is being used in the 2D space or the 3D space. For example,the 3D space may allow for interaction with the 2D space from within theHMD, and as such, the input context value may indicate that the contextis the 2D space even when the display context value may indicate the 3Dspace. In some cases, the input context may be based on tracking acurrent foreground window and determining if that window is associatedwith a 2D space or a 3D space.

Thus, the combination of input context value and display context valuemay create four operating states that the user can transition among,including:

-   -   2D input context value, 2D display context value—e.g., the user        may be at a 2D monitor interacting with 2D window content;    -   2D input context value, 3D display context value—e.g., the user        may be using the HMD while interacting with 2D window content;    -   3D input context value, 2D display context value—e.g., the user        may be at the 2D monitor interacting with 3D window content; and    -   3D input context value, 3D display context value—e.g., the user        is in the HMD interacting with 3D window content.

To determine which state to operate in, the apparatus and methods of thepresent disclosure may be used to monitor several inputs, namely, theinput context and the display context. For example, by monitoring anactive foreground window, the present disclosure may track the inputcontext value by reading the foreground window's display context value.Additionally, to determine the display context value, the presentdisclosure may monitor whether the user is actively wearing the HMD. Inthis way, the present disclosure may be used to reduce the complexity ofuser interactions and answer questions about designing the operation ofthe computing device based on the user's intentions. For example, in acomputer device with 2D and 3D interfaces, tracking the input contextand the display context to determine an operating state as described bythe present disclosure may be used to address questions such as, but notlimited to, the following:

-   -   On which display should the mouse be shown?    -   Should a pop-up appear in the 2D or 3D display?    -   When a new window is created, where should that window be        placed—in the 2D space or inside the 3D space?    -   Upon clicking a link, should the content that opens in response        to the link be placed in the 2D space or inside the 3D space?    -   When the user issues a voice command to the system, should the        command be handled in the context of the 2D space or the 3D        space?

Referring to FIGS. 1 and 2, an example computer device 102 operates inone or both of a two dimensional (2D) space 110 and a three dimensional(3D) space 120, and an operating state controller 130 of the computerdevice 102 enables a plurality of operating states including one or moreoperating states that enable interoperability between the 2D space andthe 3D space. For example, the operating states may include a 2Dinput/2D display operating state 132 (also referred to as 2D operatingstate), a 2D input/3D display operating state 134, a 3D input/2D displayoperating state 136, and a 3D input/3D display operating state 138 (alsoreferred to as 3D operating state). The 2D input/3D display operatingstate 134 and the 3D input/2D display operating state 136 provide for2D/3D interoperability.

For instance, computer device 102 may execute one or more applications14 that generates and/or interacts with one or both of 2D space 110 and3D space 120. The application 14 may be computer readable code orinstructions executable to perform a game, an office productivityapplication, or any other type of application. The 2D space 110 may bepresented on a 2D display 112, such as on a monitor of the computerdevice 102. Further, the user 103 may provide inputs into a first 2Dinput device 114, such as a keyboard in communication with the computerdevice 102 or a second 2D input device 116, such as a mouse incommunication with the computer device 102, in order to provide commandsand/or data that may be used by the one or more applications 14 in the2D space 110. The 3D space 120 may be presented to the user 103 via a 3Ddisplay 122, such as a head mounted display (HMD) 105. The HMD 105 maybe configured to provide virtual reality images (e.g., from at least onevirtual environment input), mixed reality (MR) images (e.g., from atleast two virtual environment inputs), and/or augmented reality (AR)images (e.g., from at least one virtual environment input and one realenvironment input). Also, the user 103 may provide inputs into a 3Dinput device 124, such as a virtual keyboard presented by the 3D display122, in order to provide commands and/or data that may be used by theone or more applications 14 in the 3D space 120. Moreover, in somecases, the user 103 may desire to provide a 2D input for use with the 3Ddisplay 122, and/or to provide a 3D input for use with the 2D display112.

In order to determine whether a user 103 is interacting with 2D or 3Dinterfaces, or if 2D/3D interoperability is desired, an operating statecontroller 130 may monitor a display context value 32 associated with adisplay being used by the user, and an input context value 34 associatedwith the one of the 2D space 110 or the 3D space 120 toward which a userinput is being directed. For example, the 2D operating state 132 isassociated with detecting a 2D input context and a 2D display context,the 2D input/3D display operating state 134 is associated with detectinga 2D input context and a 3D display context, the 3D input/2D displayoperating state 136 is associated with detecting a 3D input context anda 2D display context, and the 3D operating state 138 is associated withdetecting a 3D input context and a 3D display context.

In an implementation, the operating state controller 130 may be incommunication with a presence sensor 28, which monitors whether or not ahead mounted display (HMD) 105 is in use by the user 103, and changes avalue of the display context 32 accordingly. To determine whether theHMD 105 is in use, the presence sensor 28 may monitor movement of theHMD 105. For example, the presence sensor 28 may capture movement of theHMD 105 to determine whether the HMD is being placed over, for example,eyes of the user 103. This may be monitored based on HMD trackinginformation (e.g., a position and/or orientation of the HMD 105). Thus,when the user 103 puts on the HMD 105 while the HMD 105 is coupled tothe computer device 102, the presence sensor 28 may instruct theoperating state controller 130 to assign the display context value 32 a3D value.

In some implementations, when the user 103 is wearing the HMD 105 and/orwhen the display context value 32 has the 3D value, any new applicationviews may be placed into the 3D space 120, rather than being placed inthe 2D space 110.

In contrast, when the HMD 105 is not worn by the user 103, when the HMD105 is not present and/or plugged in, i.e., not coupled to the computerdevice 102, or when the user 103 removes the HMD 105, the presencesensor 28 may instruct the operating state controller 130 to assign thedisplay context value 32 a 2D value. Thus, when the user 103 is notwearing the HMD 105 and/or the display context value 32 a 2D value, anynew application views may be displayed in the 2D space 110.

In some implementations, the display context value 32 may be drivenbased on a simulation mode. For example, even when the HMD 105 is notattached (physically or wirelessly) to the computer device 102, theoperating state controller 130 may still support the 2D input/3D displayoperating state 134 based on one or more signals from an HMD simulator50 that simulates the HMD 105 being in use by the user 103 in lieu ofthe presence sensor 28 detecting such usage. In this way, the HMDsimulator 50 may be used to change the display context value 32 from 2Dto 3D even when then HMD 105 is not in use by the user 103.

In further implementations, each of the one or more applications 14 maybe assigned a display context value 32 indicating where (e.g., the 2Dspace 110 or the 3D space 120) the one or more applications 14 isrendered and/or presented to the user 103. For example, for anapplication 14 shown in the 2D space 110, the display context value 32may be a 2D value, whereas for an application 14 shown in the 3D space120, the display context value 32 may be 3D value.

Also, in an implementation, the operating state controller 130 may be incommunication with an input focus determiner 40, which monitors an inputfocus, e.g., a destination of user input into a keyboard and a mouse,and changes a value of the input context 34 accordingly. Thus, the inputcontext value 34 may identify whether the user's input focus, e.g., userinput to a mouse and/or a keyboard, is being used in the 2D space 110 orthe 3D space 120. In some implementations, this may be based on whetherthe HMD 105 is currently worn by the user 103, as determined by thepresence sensor 28. For example, when the HMD 105 is currently worn bythe user 103, the input focus determiner 40 may determine that thewindow currently focused on is in the 3D space 120. In furtherimplementations, the input focus determiner 40 may track whether thewindow currently being focused on is in the 2D space 110 or the 3D space120 based on user interactions or new programs opening. In this way, theinput focus determiner 40 monitors for changes in the foreground window,determines whether the foreground window is in the 2D space 110 or inthe 3D space 120, and sets the input context value to either a 2D or 3Dvalue accordingly. In some instances, the 3D space 120 may allow forinteraction with the 2D space from within the HMD 105, and as such, theinput context value 34 may indicate that the context may be in the 2Dspace 110 even when the display context value 32 may indicate thecontext may be in the 3D space 120.

The one or more applications 14 may be executed by a processor topresent a graphical user interface and/or images, for example, VR, MR,or AR images, on the HMD 105 or a 2D display device 112. The one or moreapplications 14 may include, for example, different VR, MR, or ARapplications (e.g., including MR and/or AR applications) that maygenerate a virtual environment in the 3D space 120. In addition, the oneor more applications 14 may provide content for the virtual environmentin the 3D space 120. For example, application 14 may provide a virtualgame for the virtual environment or provide an object that may be usedwithin virtual environment in the 3D space 120. In addition, oralternatively, the one or more applications 14 may also include, forexample, applications that display the 2D space 110 in a 2D manner,e.g., on the 2D display device 112.

In some implementations, when the user 103 starts the computer device102, the operating state controller 130 may select the 2D operatingstate 132, for example, as an initial default or based on userinteraction. That is, the user 103 may be interacting with the 2D space110 using one or more 2D inputs 114, 116 and 2D display 112. In furtherimplementations, when the user 103 starts the computer device 102 and iswearing the HMD 105, the operating state controller 130 may select the3D operating state 138. That is, the user 103 may be interacting withthe 3D space 120 using 3D input 124 and 3D display 122.

In some implementations of transitioning to the 2D input/3D displayoperating state 134, while in the 2D operating state 132, the presencesensor 28 may be used to detect the user 103 putting on the HMD 105while the HMD 105 is coupled to the computer device 102. In turn, thedisplay context value 34 may change from 2D to 3D as a result of thedisplay being utilized changing from the 2D display 112 to the 3Ddisplay 122. In this case, the operating state controller 130 may changefrom the 2D operating state 132 and select the 2D input/3D displayoperating state 134.

In some cases of transitioning to the 2D input/3D display operatingstate 134, the presence sensor 28 may be used to detect the user 103putting on the HMD 105 and the input focus determiner 40 may detect theuser 103 using an application 14 in the HMD 105, and as a result, maycause the computer device 102 may assign the foreground to the 3D space120. For example, if the user 103 is using a word processing application14 in the 2D space 110, the computer device 102 may automatically shiftthe foreground from the 2D display 112 to the 3D display 122. As aresult, any keystrokes using the 2D input 114 or movements using the 2Dinput 116 may be directed to the application 14 in the 3D space 120. Asa result, the operating state controller 130 selects the 2D input/3Ddisplay operating state 134.

In some instances of transitioning to the 3D operating state 138, whenfocus is assigned to the 3D space 120 and the HMD 105 is put on by theuser 103, the input context value 34 may be automatically changed from2D to 3D to match the display context value 32. As a result, theoperating state controller 130 selects the 3D operating state 138.

As another example of transitioning to the 2D input/3D display operatingstate 134, when the focus of the user 103 is in the 3D space 120, theinput context value 34 may be 3D. However, the computer device 102 mayexecute an operation in the 2D space 110 when the user 103 is using theHMD 105 that causes the foreground to shift from the HMD 105 to the 2Ddisplay 112. As a result, the input focus determiner 40 may shift theinput context value 34 from 3D to 2D, even though the display contextvalue 32 is still 3D. As a result, the operating state controller 130selects the 2D input/3D display operating state 134. In this case, thecomputer device 102 may cause the application 14 to open in the HMD 105,rather than the 2D display 112, in response to opening 2D window whilethe user 103 is wearing the HMD 105.

In a further example of transitioning to the 3D input/2D displayoperating state 136, the presence sensor 28 may detect when the user 103removes the HMD 105, and in response, the presence sensor 28 may notifythe other components of the computer device 102 of such removal and thatthe display context value 32 has changed from 3D to 2D. As a result, theoperating state controller 130 selects the 3D input/2D display operatingstate 136. In some implementations, the input focus determiner 40 mayautomatically cause the input context value 34 to change from 3D to 2Dwhen the user 103 removes the HMD 105, thereby automatically changingthe operating state to the 2D operating state 132. However, in somecases, the user 103 may be using the 2D display 112 while debugging a 3Dapplication 14. In such cases, the input context value 34 may be remain,or may be changed back to, 3D. That is, in such an example, the inputcontext value may be 3D, while the display context value may be 2D. As aresult, the operating state controller 130 may select the 3D input/2Ddisplay operating state 136.

In further implementations, the computer device 102 may also include atoggle input detector 150, which may be configured to detect when theuser provides a toggle input, such as entering a hotkey or a sequence ofor combination of keys, to change or switch the display context valueand/or the input context value. In effect, the toggle input may simulatethe user 103 taking off or putting on the HMD 105, and may or may notchange in the input context.

FIG. 5 illustrates transitions between operating states in response toreceiving a toggle input in accordance with an implementation of thepresent disclosure. For example, as illustrated in FIG. 5, when thedisplay context value and the input context value 34 are both 2D,receiving the toggle input simulates the HMD 105 being put on by theuser 103 and moves the keyboard focus to the 3D space 120. As anotherexample, as illustrated in FIG. 5, when the display context value 32 is2D and the input context value 34 is 3D, receiving the toggle inputsimulates the HMD 105 being put on by the user 103 and leaves keyboardfocus as-is. In yet another example, as illustrated in FIG. 5, when thedisplay context value 32 is 3D and the input context value 34 is 2D,receiving the toggle input simulates the HMD 105 being taken off by theuser 103 and moves the keyboard focus to the 3D space 120. In yetanother example, as illustrated in FIG. 5, when the display contextvalue 32 and the input context value are both 3D, receiving the toggleinput simulates the HMD 105 being taken off by the user 103 and movesthe keyboard focus to the 2D space 110.

In further implementations, when the computer device 102 is operating inany one of the 2D input, 3D display operating state 134, the 3D input,2D display operating state 136, or the 3D operating state 138, thetoggle input detector 150 may detect the user 103 entering the toggleinput, and as a result, the operating state controller 130 may changethe operating state back to the 2D operating state 132. That is, in somecases, receiving the toggle input may cause the computer device 102 toselect a 2D window to receive focus and to simulate, if necessary,removing the HMD 105. In still further implementations, when thecomputer device 102 is operating in the 2D operating state 132, thetoggle input detector 150 may detect the user 103 entering the toggleinput, and as a result, the operating state controller 130 may changethe operating state to the 3D operating state 138.

The computer device 102 may also include a memory configured for storingdata and/or computer-executable instructions defining and/or associatedwith an operating system 106, and a processor that may execute theoperating system. As an example, the memory can include, but is notlimited to, a type of memory usable by a computer, such as random accessmemory (RAM), read only memory (ROM), tapes, magnetic discs, opticaldiscs, volatile memory, non-volatile memory, and any combinationthereof. As an example, the processor can include, but is not limitedto, any processor specially programmed as described herein, including acontroller, microcontroller, application specific integrated circuit(ASIC), field programmable gate array (FPGA), system on chip (SoC), orother programmable logic or state machine. Processor and/or memory mayexecute code that defines the functionality described herein to controlan operating state of the computer device 102 and, in some cases, enable2D/3D interoperability.

The computer device 102 may include any mobile or fixed computer device,which may be connectable to a network. The computer device 102 may be,for example, a computer device such as a desktop or laptop or tabletcomputer, a cellular telephone, a gaming device, a mixed reality orvirtual reality device, a music device, a television, a navigationsystem, a camera, a personal digital assistant (PDA), or a handhelddevice, or any other computer device having wired and/or wirelessconnection capability with one or more other devices and/orcommunication networks.

Thus, according to the present disclosure, computer device 102 isconfigured to enable interoperability between 2D and 3D interfaces,thereby unifying 2D and 3D operations on computer device 102.

As shown with respect to FIG. 2, the HMD 105 may include a headpiece110, which may be a headband, arranged to be worn on the head of user,e.g., user 103 of FIG. 1. It should be appreciated by those of ordinaryskill in the art that the HMD 105 may also be attached to the user'shead using a frame (in the manner of conventional spectacles), helmet,or other fit system. The purpose of the fit system is to support thedisplay and provide stability to the display and other head-bornesystems such as tracking systems, sensors, and cameras. The HMD 105 mayalso include optical components 215 (e.g., one or more lenses),including waveguides that may allow the HMD 105 to project imagesgenerated by a light engine included within (or external) to the HMD105. The optical components 215 may use plate-shaped (usually planar)waveguides for transmitting angular image information to users' eyes asvirtual images from image sources located out of the user's line ofsight. The image information may propagate along the waveguides as aplurality of angularly related beams that are internally reflected alongthe waveguide. Diffractive optics are often used for injecting the imageinformation into the waveguides through a first range of incidenceangles that are internally reflected by the waveguides as well as forejecting the image information through a corresponding range of lowerincidence angles for relaying or otherwise forming an exit pupil behindthe waveguides in a position that can be aligned with the users' eyes.Both the waveguides and the diffractive optics at the output end of thewaveguides may be at least partially transparent so that the user 103can also view the real environment through the waveguides, such as whenthe image information is not being conveyed by the waveguides or whenthe image information does not fill the entire field of view.

The light engine (not shown), which may project images to be displayedon the optical components 215, may comprise a light source (e.g., alaser), a micro display and imaging optics in the form of a collimatinglens. The micro display can be any type of image source, such as liquidcrystal on silicon (LCoS) displays, liquid crystal displays (LCD),matrix arrays of LED's (whether organic or inorganic) and any othersuitable display. The optical components 215 may focus a user's visionon one or more portions of one or more display panels. The displaypanels may display one or more images (e.g., left eye image and righteye image) based on signals received from the light engine. Thus, theoptics may include left eye optics for focusing the user's left eye onthe left eye image and right eye optics for focusing the user's righteye on the right eye image. For example, optical components 215 mayfocus the user's eyes on a central portion of each of the left eye imageand the right eye image. The user's brain may combine the images viewedby each eye to create the perception that the user 103 is viewing a 3Denvironment.

The HMD 105 may also include a frame or slot (not shown) for coupling asmartphone or other portable computing device to the HMD 105 in a waythat exposes the display screen of the smartphone to optical components215. The HMD 105 may transition between AR, MR, and VR scenarios bycausing the optical members to either pass light or block light, therebyeither showing or hiding view of the real world while presenting virtualcontent to the user 103.

The computer device 102 may execute the following sample actions whenthe display context value 32 indicates that the HMD 105 is beingutilized:

-   -   auto-launching a desktop application in the HMD 105 even though        the input context value 34 indicates a 2D space;    -   disabling features of a virtual assistant;    -   enabling a 3D motion controller input; and    -   hiding a desktop mouse cursor.

The computer device 102 may also execute the following sample actionswhen the display context value 32 indicates that the 2D display 112 isbeing utilized:

-   -   disabling a gamepad input;    -   disabling a holographic keyboard; and    -   disabling 3D components from stealing foreground from 2D space        windows.

The computer device 102 execute may execute the following sample actionswhenever the display context value 32 changes:

-   -   changing how secondary tiles API are handled;    -   changing how web authentication brokers work, e.g., selecting a        holographic version for HMD;    -   changing rendering and logic for system notifications;    -   changing banner visibility;    -   changing where a digital assistant will appear;    -   changing which file picker is used;    -   determining how the system should behave while using a simulated        HMD;    -   hiding a “soft” keyboard on the desktop;    -   changing how speech APIs interpret speech;    -   determining whether the 2D space reacts to speech input; and    -   ignoring windows displayed in the HMD.

The computer device 102 may execute the following sample actions whenthe input context value 34 indicates that the 2D display 112 is beingutilized:

-   -   auto-launching the desktop application in the HMD 105.

The computer device 102 may execute the following sample actions whenthe input context value 34 indicates that the HMD 105 is being utilized:

-   -   disabling hotkeys in a browser window; and    -   disabling game bar hotkeys and other shortcuts.

The computer device 102 execute may execute the following sample actionswhenever the input context value 34 changes:

-   -   toggling visibility of a holographic input banner, e.g., a        notification that a keyboard sequence can be pressed to use a        mouse on the 2D display 112 while the user 103 is wearing the        HMD 105.

It should be appreciated by those of ordinary skill in the art that theaforementioned sample actions are merely example actions that areperformed by the computer device 102 and that other actions may also beperformed by the computer device 102 based on the display context valueand the input context value.

Referring now to FIG. 3, an example method 300 for controlling anoperating state that may be used by the computer device 102 (FIG. 1).

At 305, the method 300 may include identifying a display context valuethat indicates whether a user is viewing a two dimensional (2D) displayof a 2D space or a three dimensional (3D) display of a 3D space of thecomputer device. For example, in an implementation, computer device 102may operate a presence sensor 28 (FIG. 1) to identify a display contextvalue that indicates whether a user is viewing a two dimensional (2D)display of a 2D space or a three dimensional (3D) display of a 3D spaceof the computer device.

At 310, the method 300 may include identifying an input context valuethat indicates whether a user input received at an input device isdirected toward the 2D space or the 3D space of the computer device. Forexample, the computer device 102 may execute an input focus determiner40 (FIG. 1) to a identify an input context value that indicates whethera user input received at an input device is directed toward the 2D spaceor the 3D space of the computer device.

In some implementations, the display context value may be based on useof a head-mounted display (HMD) by the user and the input context valuemay be based on whether the input from a keyboard and/or mouse isdirected to the 2D space or the 3D space. The display context value mayalso indicate the 3D display is being used when the HMD is in use, andthe display context value may indicate the 2D display is being used whenthe HMD is not in use.

In further implementations, the method may also include changing fromeither a 2D input/2D display operating state to the 2D input/3D displayoperating state, or from the 3D input/2D display operating state to a 3Dinput/3D display operating state, when the HMD is detected as being inuse while the HMD is coupled to the computer device. The method may alsoinclude changing from either the 2D input/3D display operating state tothe 2D input/2D display operating state, or from the 3D input/3D displayoperating state to the 3D input/2D display operating state, when the HMDis detected as not being in use while the HMD is coupled to the computerdevice.

In some implementations, identifying the input context value may includeidentifying an active foreground window, identifying a 2D value or a 3Dvalue of the display context of the active foreground window, andsetting the input context value to match the 2D value or the 3D value ofthe display context.

At 315, the method 300 may further include selecting one of a pluralityof operating states of the computer device based on the display contextvalue and the input context value. For example, computer device 102 mayexecute operating state controller 130 (FIG. 1) to select one of aplurality of operating states of the computer device based on thedisplay context value and the input context value, where the pluralityof operating states include 2D/3D interoperability operating states. Insome implementations, for example, the plurality of operating statesinclude a 2D input/3D display operating state and a 3D input/2D displayoperating state. In some implementations, the operating state isselected such that the input is selected to match the identified inputcontext value and the display is selected to match the identifieddisplay context value.

At 320, the method 300 may further include operating the computer deviceaccording to the selected operating state. For example, in animplementation, computer device 102 may execute processor 56 (FIG. 4) tooperate the computer device 102 according to the selected operatingstate. In other words, processor 56 receives 2D or 3D inputs andgenerates 2D or 3D outputs based on the corresponding input context anddisplay context of the selected operating state.

In some instances, the operating state of the computer device 102 may bechanged from the selected operating state to a different one of theplurality of operating states. For example, the method may furtherinclude changing from either a 2D input/2D display operating state tothe 2D input/3D display operating state, or from the 3D input/2D displayoperating state to a 3D input/3D display operating state, when the HMDis detected as being in use while the HMD is coupled to the computerdevice. The method may also include changing from either the 2D input/3Ddisplay operating state to the 2D input/2D display operating state, orfrom the 3D input/3D display operating state to the 3D input/2D displayoperating state, when the HMD is detected as not being in use while theHMD is coupled to the computer device.

In still further implementations, the method may include receiving atoggle input from the user to change the operating state of the computerdevice and changing the selected one of the plurality of operatingstates in response to the toggle input. For example, the method mayinclude changing the selected operating state from the 2D input/3Ddisplay operating state, the 3D input/2D display operating state, or a3D input/3D display operating state to a 2D input/2D display operatingstate upon receiving the toggle input from the user. As another example,the method may include changing the selected operating state from the 2Dinput/2D display operating state or the 3D input/2D display to the 3Dinput/3D display operating state upon receiving the toggle input fromthe user

Referring now to FIG. 4, illustrated is an example computer device 102in accordance with an implementation, including additional componentdetails as compared to FIG. 1. As noted above, in some examples,components illustrated as part of the computer device 102 may also beimplemented directly within the display device 105. In such situations,the term “computer device” may also refer to stand alone displaydevice(s) such as, desktops, laptops, smart phones, or tablets. In oneexample, the computer device 102 may include processor 56 for carryingout processing functions associated with one or more of components andfunctions described herein. Processor 56 can include a single ormultiple set of processors or multi-core processors. Moreover, processor56 can be implemented as an integrated processing system and/or adistributed processing system.

The computer device 102 may further include memory 58, such as forstoring local versions of applications being executed by processor 56.Memory 58 can include a type of memory usable by a computer, such asrandom access memory (RAM), read only memory (ROM), tapes, magneticdiscs, optical discs, volatile memory, non-volatile memory, and anycombination thereof. Additionally, processor 56 and memory 58 mayinclude and execute operating system 106 (FIG. 1).

Further, computer device 102 may include a communications component 61that provides for establishing and maintaining communications with oneor more parties utilizing hardware, software, and services as describedherein. Communications component 61 may carry communications betweencomponents on the computer device 102, as well as between the computerdevice 102 and external devices, such as devices located across acommunications network and/or devices serially or locally connected tothe computer device 102, such as HMD 105 (FIG. 2). For example,communications component 61 may include one or more buses, and mayfurther include transmit chain components and receive chain componentsassociated with a transmitter and receiver, respectively, operable forinterfacing with external devices.

Additionally, the computer device 102 may also include a presence sensor28, which monitors whether or not a head mounted display (HMD) 105 is inuse by the user 103. To determine whether the HMD 105 is in use, thepresence sensor 28 may monitor movement of the HMD 105. For example, thepresence sensor 28 may capture movement of the HMD 105 to determinewhether the HMD is being placed over, for example, eyes of the user 103.This may be monitored based on HMD tracking information (e.g., aposition and/or orientation of the HMD 105).

The computer device 102 may also include an input focus determiner 40,which monitors an input focus, e.g., a keyboard and a mouse, and changesa value of the input context 34 accordingly. In some instances, theinput focus determiner 40 may determine whether a window currentlyfocused on is in the 2D space 110 or the 3D space 120. For example, theinput focus determiner 40 may track whether the window currently beingfocused on is in the 2D space 110 or the 3D space 120 based on userinteractions or new programs opening. In this way, the input focusdeterminer 40 monitors for changes in the foreground window, determineswhether the foreground window is in the 2D space 110 or in the 3D space120, and sets the input context value to either a 2D or 3D valueaccordingly. In some instances, the 3D space 120 may allow forinteraction with the 2D space from within the HMD, and as such, theinput context value 34 may indicate that the context may be in the 2Dspace 110 even when the display context value 32 may indicate thecontext may be in the 3D space 120.

In further implementations, the computer device 102 may also include atoggle input detector 150, which may be configured to detect when theuser provides a toggle input, such as entering a hotkey or a sequence ofor combination of keys, to change or switch the display context valueand/or the input context value.

The computer device 102 may also include an operating state controller130 that enables a plurality of operating states, namely a 2D input/2Ddisplay operating state 132 (also referred to as 2D operating state), a2D input/3D display operating state 134, a 3D input/2D display operatingstate 136, and a 3D input/3D display operating state 138 (also referredto as 3D operating state). In some implementations, the 2D input/3Ddisplay operating state 134 and the 3D input/2D display operating state136 provide for 2D/3D interoperability operating states.

The computer device 102 may also include a user interface component 64operable to receive inputs from a user 103 of the computer device 102and further operable to generate outputs for presentation to the user103. User interface component 64 may include one or more input devices,including but not limited to a keyboard, a number pad, a mouse, atouch-sensitive display, a navigation key, a function key, a microphone,a voice recognition component, any other mechanism capable of receivingan input from a user, or any combination thereof. Further, userinterface component 64 may include one or more output devices, includingbut not limited to a display, a speaker, a haptic feedback mechanism, aprinter, any other mechanism capable of presenting an output to a user103, or any combination thereof. In an implementation, user interfacecomponent 64 may transmit and/or receive messages corresponding to theoperation of applications 14.

As used in this application, the terms “component,” “system” and thelike are intended to include a computer-related entity, such as but notlimited to hardware, firmware, a combination of hardware and software,software, or software in execution. For example, a component may be, butis not limited to being, a process running on a processor, a processor,an object, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on acomputer device and the computer device can be a component. One or morecomponents can reside within a process and/or thread of execution and acomponent may be localized on one computer and/or distributed betweentwo or more computers. In addition, these components can execute fromvarious computer readable media having various data structures storedthereon. The components may communicate by way of local and/or remoteprocesses such as in accordance with a signal having one or more datapackets, such as data from one component interacting with anothercomponent in a local system, distributed system, and/or across a networksuch as the Internet with other systems by way of the signal.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Various implementations or features may have been presented in terms ofsystems that may include a number of devices, components, modules, andthe like. It is to be understood and appreciated that the varioussystems may include additional devices, components, modules, etc. and/ormay not include all of the devices, components, modules etc. discussedin connection with the figures. A combination of these approaches mayalso be used.

The various illustrative logics, logical blocks, and actions of methodsdescribed in connection with the embodiments disclosed herein may beimplemented or performed with a specially-programmed one of a generalpurpose processor, a digital signal processor (DSP), an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA) or other programmable logic device, discrete gate or transistorlogic, discrete hardware components, or any combination thereof designedto perform the functions described herein. A general-purpose processormay be a microprocessor, but, in the alternative, the processor may beany conventional processor, controller, microcontroller, or statemachine. A processor may also be implemented as a combination ofcomputer devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration. Additionally, at leastone processor may comprise one or more components operable to performone or more of the steps and/or actions described above.

Further, the steps and/or actions of a method or algorithm described inconnection with the implementations disclosed herein may be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a harddisk, a removable disk, a CD-ROM, or any other form of storage mediumknown in the art. An exemplary storage medium may be coupled to theprocessor, such that the processor can read information from, and writeinformation to, the storage medium. In the alternative, the storagemedium may be integral to the processor. Further, in someimplementations, the processor and the storage medium may reside in anASIC. Additionally, the ASIC may reside in a user terminal. In thealternative, the processor and the storage medium may reside as discretecomponents in a user terminal. Additionally, in some implementations,the steps and/or actions of a method or algorithm may reside as one orany combination or set of codes and/or instructions on a machinereadable medium and/or computer readable medium, which may beincorporated into a computer program product.

In one or more implementations, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored or transmittedas one or more instructions or code on a computer-readable medium.Computer-readable media includes both computer storage media andcommunication media including any medium that facilitates transfer of acomputer program from one place to another. A storage medium may be anyavailable media that can be accessed by a computer. By way of example,and not limitation, such computer-readable media can comprise RAM, ROM,EEPROM, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium that can be used tostore desired program code in the form of instructions or datastructures and that can be accessed by a computer. Disk and disc, asused herein, includes compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk and Blu-ray disc where disksusually reproduce data magnetically, while discs usually reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

While implementations of the present disclosure have been described inconnection with examples thereof, it will be understood by those skilledin the art that variations and modifications of the implementationsdescribed above may be made without departing from the scope hereof.Other implementations will be apparent to those skilled in the art froma consideration of the specification or from a practice in accordancewith examples disclosed herein.

What is claimed is:
 1. A method of operating a computer device,comprising: identifying a display context value in response to usage ofa head-mounted display (HMD) that indicates that a user is viewing a twodimensional (2D) display of a 2D space of the computer device inresponse to the HMD not being in use or that indicates that the user isviewing a three dimensional (3D) display of a 3D space of the computerdevice in response to the HMD being in use; identifying an input contextvalue that indicates whether a user input received at an input device isdirected toward the 2D space or the 3D space of the computer device;selecting one of a plurality of operating states of the computer devicebased on the display context value and the input context value, theplurality of operating states including a 2D input/3D display operatingstate and a 3D input/2D display operating state; and operating thecomputer device according to the selected operating state.
 2. The methodof claim 1, wherein: the input context value is based on whether theuser input from a keyboard and/or mouse is directed to the 2D space orthe 3D space.
 3. The method of claim 2, further comprising: changingfrom either a 2D input/2D display operating state to the 2D input/3Ddisplay operating state, or from the 3D input/2D display operating stateto a 3D input/3D display operating state, when the HMD is detected asbeing in use while the HMD is coupled to the computer device; andchanging from either the 2D input/3D display operating state to the 2Dinput/2D display operating state, or from the 3D input/3D displayoperating state to the 3D input/2D display operating state, when the HMDis detected as not being in use while the HMD is coupled to the computerdevice.
 4. The method of claim 1, wherein identifying the input contextvalue comprises: identifying an active foreground window; identifying a2D value or a 3D value of the display context value of the activeforeground window; and setting the input context value to match the 2Dvalue or the 3D value of the display context value.
 5. The method ofclaim 1, further comprising: receiving a toggle input from the user tochange the operating state of the computer device; and changing theselected operating state in response to the toggle input.
 6. The methodof claim 5, further comprising: changing the selected operating statefrom the 2D input/3D display operating state, the 3D input/2D displayoperating state, or a 3D input/3D display operating state to a 2Dinput/2D display operating state upon receiving the toggle input fromthe user; and changing the selected operating state from the 2D input/2Ddisplay operating state or the 3D input/2D display to the 3D input/3Ddisplay operating state upon receiving the toggle input from the user.7. A computer device for selecting an operating state, comprising: amemory to store data and instructions; and a processor in communicationwith the memory to execute the instructions to: identify a displaycontext value in response to usage of a head-mounted display (HMD) thatindicates that a user is viewing a two dimensional (2D) display of a 2Dspace of the computer device in response to the HMD not being in use orthat indicates that the user is viewing a three dimensional (3D) displayof a 3D space of the computer device in response to the HMD being inuse; identify an input context value that indicates whether a user inputreceived at an input device is directed toward the 2D space or the 3Dspace of the computer device; select one of a plurality of operatingstates of the computer device based on the display context value and theinput context value, the plurality of operating states including a 2Dinput/3D display operating state and a 3D input/2D display operatingstate; and operate the computer device according to the selectedoperating state.
 8. The computer device of claim 7, wherein: the inputcontext value is based on whether the user input from a keyboard and/ormouse is directed to the 2D space or the 3D space.
 9. The computerdevice of claim 8, wherein the instructions are further executable to:change from either a 2D input/2D display operating state to the 2Dinput/3D display operating state, or from the 3D input/2D displayoperating state to a 3D input/3D display operating state, when the HMDis detected as being in use while the HMD is coupled to the computerdevice; and change from either the 2D input/3D display operating stateto the 2D input/2D display operating state, or from the 3D input/3Ddisplay operating state to the 3D input/2D display operating state, whenthe HMD is detected as not being in use while the HMD is coupled to thecomputer device.
 10. The computer device of claim 7, wherein identifyingthe input context value comprises: identify an active foreground window;identify a 2D value or a 3D value of the display context value of theactive foreground window; and set the input context value to match the2D value or the 3D value of the display context value.
 11. The computerdevice of claim 10, wherein the instructions are further executable to:receive a toggle input from the user to change the operating state ofthe computer device; and change the selected operating state in responseto the toggle input.
 12. The computer device of claim 11, wherein:changing the selected operating state from the 2D input/3D displayoperating state, the 3D input/2D display operating state, or a 3Dinput/3D display operating state to a 2D input/2D display operatingstate upon receiving the toggle input from the user; and changing theselected operating state from the 2D input/2D display operating state orthe 3D input/2D display to the 3D input/3D display operating state uponreceiving the toggle input from the user.
 13. A non-transitorycomputer-readable medium storing instructions executable by a computerdevice, the instructions comprising code for: identifying a displaycontext value in response to usage of a head-mounted display (HMD) thatindicates that a user is viewing a two dimensional (2D) display of a 2Dspace of the computer device in response to the HMD not being in use orthat indicates that the user is viewing a three dimensional (3D) displayof a 3D space of the computer device in response to the HMD being inuse; identifying an input context value that indicates whether a userinput received at an input device is directed toward the 2D space or the3D space of the computer device; selecting one of a plurality ofoperating states of the computer device based on the display contextvalue and the input context value, the plurality of operating statesincluding a 2D input/3D display operating state and a 3D input/2Ddisplay operating state; and operating the computer device according tothe selected operating state.
 14. The non-transitory computer-readablemedium of claim 13, wherein: the input context value is based on whetherthe user input from a keyboard and/or mouse is directed to the 2D spaceor the 3D space.
 15. The non-transitory computer-readable medium ofclaim 14, the instructions further comprising code for: changing fromeither a 2D input/2D display operating state to the 2D input/3D displayoperating state, or from the 3D input/2D display operating state to a 3Dinput/3D display operating state, when the HMD is detected as being inuse while the HMD is coupled to the computer device; and changing fromeither the 2D input/3D display operating state to the 2D input/2Ddisplay operating state, or from the 3D input/3D display operating stateto the 3D input/2D display operating state, when the HMD is detected asnot being in use while the HMD is coupled to the computer device. 16.The non-transitory computer-readable medium of claim 13, theinstructions further comprising code for: identifying an activeforeground window; identifying a 2D value or a 3D value of the displaycontext value of the active foreground window; and setting the inputcontext value to match the 2D value or the 3D value of the displaycontext value.
 17. The non-transitory computer-readable medium of claim13, the instructions further comprising code for: receiving a toggleinput from the user to change the operating state of the computerdevice; changing the selected operating state from the 2D input/3Ddisplay operating state, the 3D input/2D display operating state, or a3D input/3D display operating state to a 2D input/2D display operatingstate upon receiving the toggle input from the user; and changing theselected operating state from the 2D input/2D display operating state orthe 3D input/2D display to the 3D input/3D display operating state uponreceiving the toggle input from the user.
 18. A method of operating acomputer device, comprising: identifying a display context value thatindicates whether a user is viewing a two dimensional (2D) display of a2D space or a three dimensional (3D) display of a 3D space of thecomputer device, wherein the display context value is based on use of ahead-mounted display (HMD) by the user; identifying an input contextvalue that indicates whether a user input received at an input device isdirected toward the 2D space or the 3D space of the computer device,wherein the input context value is based on whether the user input froma keyboard or mouse is directed to the 2D space or the 3D space;selecting one of a plurality of operating states of the computer devicebased on the display context value and the input context value, theplurality of operating states including a 2D input/3D display operatingstate and a 3D input/2D display operating state; operating the computerdevice according to the selected operating state; changing from either a2D input/2D display operating state to the 2D input/3D display operatingstate, or from the 3D input/2D display operating state to a 3D input/3Ddisplay operating state, when the HMD is detected as being in use whilethe HMD is coupled to the computer device; and changing from either the2D input/3D display operating state to the 2D input/2D display operatingstate, or from the 3D input/3D display operating state to the 3Dinput/2D display operating state, when the HMD is detected as not beingin use while the HMD is coupled to the computer device.
 19. A computerdevice for selecting an operating state, comprising: a memory to storedata and instructions; and a processor in communication with the memoryto execute the instructions to: identify a display context value thatindicates whether a user is viewing a two dimensional (2D) display of a2D space or a three dimensional (3D) display of a 3D space of thecomputer device, wherein the display context value is based on use of ahead-mounted display (HMD) by the user; identify an input context valuethat indicates whether a user input received at an input device isdirected toward the 2D space or the 3D space of the computer device,wherein the input context value is based on whether the user input froma keyboard or mouse is directed to the 2D space or the 3D space; selectone of a plurality of operating states of the computer device based onthe display context value and the input context value, the plurality ofoperating states including a 2D input/3D display operating state and a3D input/2D display operating state; operate the computer deviceaccording to the selected operating state; change from either a 2Dinput/2D display operating state to the 2D input/3D display operatingstate, or from the 3D input/2D display operating state to a 3D input/3Ddisplay operating state, when the HMD is detected as being in use whilethe HMD is coupled to the computer device; and change from either the 2Dinput/3D display operating state to the 2D input/2D display operatingstate, or from the 3D input/3D display operating state to the 3Dinput/2D display operating state, when the HMD is detected as not beingin use while the HMD is coupled to the computer device.
 20. Anon-transitory computer-readable medium storing instructions executableby a computer device, the instructions comprising code for: identifyinga display context value that indicates whether a user is viewing a twodimensional (2D) display of a 2D space or a three dimensional (3D)display of a 3D space of the computer device, wherein the displaycontext value is based on use of a head-mounted display (HMD) by theuser; identifying an input context value that indicates whether a userinput received at an input device is directed toward the 2D space or the3D space of the computer device, wherein the input context value isbased on whether the user input from a keyboard or mouse is directed tothe 2D space or the 3D space; selecting one of a plurality of operatingstates of the computer device based on the display context value and theinput context value, the plurality of operating states including a 2Dinput/3D display operating state and a 3D input/2D display operatingstate; operating the computer device according to the selected operatingstate; changing from either a 2D input/2D display operating state to the2D input/3D display operating state, or from the 3D input/2D displayoperating state to a 3D input/3D display operating state, when the HMDis detected as being in use while the HMD is coupled to the computerdevice; and changing from either the 2D input/3D display operating stateto the 2D input/2D display operating state, or from the 3D input/3Ddisplay operating state to the 3D input/2D display operating state, whenthe HMD is detected as not being in use while the HMD is coupled to thecomputer device.